BACKGROUND & PURPOSE:

Keeping up with the rapid pace of change in the healthcare system and the development of technology has dictated that clinicians learn about ways to improve quality of care over the course of their careers. Yet there has been little study of the association between the process of rehabilitation education and quality care.[1]  

Assessing training effectiveness is complex and costly. There is fundamental difficulty in addressing the questions that need to be answered: what works, in what context, with which groups, and at what cost? Additionally, there are few proven methodologies.

The length of time needed for the evaluation, lag time between an educational intervention and follow up evaluation, lack of reliable objective measures, and the number of potential confounding factors increase the complexity of the issue under study. Challenges designing methodologies that can control for variations in training programs are vast. Variations include clinician knowledge, skills, and training; patient comorbidities and differences in severity of illness, and system level variables, such as policies and regulations influencing patient care practices and funding. For these reasons, health professionals are often reluctant to study the effectiveness of educational interventions.

Consequently, it is not surprising that research validating effective methods to train clinicians, influence practice patterns or impact patient outcomes is lacking [2]. Systematic reviews [3-5] of the educational literature found that  few robust evaluations of educational interventions exist.  However, some studies concluded that continuing education can improve clinical performance and patient outcomes, and indicated which methods were best at evoking change in clinician behavior. The most effective methods derived from these reviews include learning linked to clinical practice, interactive educational meetings, outreach events and strategies that involve multiple educational interventions (i.e. outreach plus reminders).[3-6] The least effective methods are also the most commonly used in general practice medical education- namely, lecture format teaching and unsolicited printed material such as clinical guidelines. [7]

The objective of this study is to measure the utilization of web-based rehabilitation research training by measuring short- and mid-term impacts on knowledge and attitudes of clinicians. In particular, this study examines the effect of a web-based training (WBT) program on clinician knowledge of manual wheelchair technology, and attitudes towards practice for professionals recommending seating and wheeled mobility equipment.

METHODOLOGY:

Training Intervention 

A web-based training (WBT) intervention tailored for clinicians responsible for recommending manual wheelchair technologies was designed synthesizing “best practice” and “state of the science” research literature pertaining to seating and mobility for manual wheelchair users. A pretest-post test design with control group was employed using a convenience sample (39 WBT, 28 Control). Two measures designed to detect change in clinical knowledge, and attitudes were administered before, after and 6 months following the intervention. The control group completed the knowledge measure at time of initial contact, and 6 months later.

Evaluation Criteria  

The upper levels of Kirkpatrick’s hierarchy for assessing training effectiveness were the foundation for developing two measures. Specifically, we were interested in learning how clinical practices recommending and specifying manual wheelchairs for clients with mobility impairments change following an educational training program.

Kirkpatrick’s level 2 (knowledge) was the basis for developing the Knowledge Questionnaire. A 15-question multiple-choice test assessing knowledge of empirical research and “best practices” as related to manual wheelchair applications was administered before (Pretest), immediately after (Posttest), and 6 months following the WBT program (Follow-up). The WBT Knowledge Questionnaire included eight questions that were used with the control group. These like items were used to compare the WBT and Control groups.

A Manual Wheelchair Practice Questionnaire was used to explore Kirkpatrick’s level 3 (transfer). This level is intended to measure the transfer that has occurred in a learner’s attitudes due to a WBT program. Evaluation at this level attempts to answer the question, “Is the newly acquired attitude being used in everyday clinical practice?” Questions were asked about recommending and specifying manual wheelchair equipment and the attitudes towards practice while doing so. Items were grouped into four domains: Independence, Confidence, Leadership, and Resourcefulness. We explored whether changes in these four domains could be detected immediately following the web-course intervention and, if so, whether or not a change persisted 6 months afterward.

Study Enrollment  

A total of 40 subjects were consented and enrolled in the WBT study and 39 completed the study. One subject withdrew from the study due to personal reasons. The WBT group was followed for 6 months after the intervention. Historical data from 26 control subjects from an earlier study examining the effects of a traditional continuing education program were used for control group comparisons.

RESULTS AND DISCUSSION:

Demographics

Results did show a significant difference in major occupations between the WBT ( PTs = 20, OTs = 18) and Control group (PTs = 26, OTs = 0). There are recognized regional differences in practice; in some regions PTs are responsible for seating and mobility service provision and in other locations OTs are the lead providers. The WBT group has representation from 19 states whereas the Control group represents only 1state perhaps accounting for the difference in professional distribution. There were no statistically significant differences between the WBT and Control groups for years of clinical practice, years of seating and mobility, hours of seating and mobility service or professional development hours. The largest difference in the groups was in the number of hours per week providing seating and mobility services. Although not significant, the WBT group performed an average of 9.8 hours whereas the Control group performed 4.9 hours. (Figure 1)

Knowledge Scores of WBT and Control Groups 

The results of a repeated measures ANOVA showed no interaction between Group (Control and WBT) and repeated Knowledge test scores (Pre-test and 6 month Follow-up), (partial Eta squared of .003 and observed power of .070). These results indicate that changes in Knowledge scores over time did not vary across the WBT and Control Groups. In fact, both groups showed a similar response over time when considering the 8-item test. Lack of power may have masked differences between Control and WBT groups.

The results of a repeated measures ANOVA for the full WBT 15-item Knowledge test showed differences in scores over time (df=2, F=10.203, p=.000, partial Eta squared= .212 and observed power of .984). Posttest and Follow-up test scores were higher than Pretest scores. Follow-up scores showed a decreasing trend compared to Posttest values (p=0.055) but remained above baseline. (Figure 2) We do not know if knowledge plateaus or continues to decline over time. A longer follow along period is an opportunity for future study.

Changes in Attitude of WBT Group 

The MWC Practice Questionnaire assessed attitude in four domains: Confidence, Independence, Leadership and Resourcefulness. WBT participants took the questionnaire at the Pretest, Posttest and Follow-up time periods. Control subjects reported attitudes twice representing the Pretest and Follow-up time periods.

Attitude scores from the Control group within all four domains were not different between Pretest and Follow-up (P>0.3). Individual ANOVA for the WBT group combined with Tukey analysis was used for post-hoc comparisons of each timeframe and significance reported for p<0.05.

The WBT group reported greater feelings of Independence and Leadership but no change in Confidence or Resourcefulness. In other words, 6 months after web-based training, clinicians reported a greater sense of Leadership and Independence but did not use any more resources or feel more confident in selecting products.

Leadership was measured using 3 items that asked the percentage of time that clinicians took the lead role in selecting and choosing features and equipment and authoring the letter of medical necessity. For the WBT group, the sense of Leadership was higher after six months indicating that the training was effective in instilling this attitude.

Three Independence items queried therapists about their senses of Independence while recommending and specifying manual wheelchair equipment. Feelings of Independence rose immediately after WBT and continued to rise over the 6-month follow-up period.

The Resourcefulness domain included questions about the resources used by clinicians as they recommended and specified manual wheelchair equipment. Items asked about the percentage of time working with a certified supplier, reasons for relying on the supplier, resources used in the previous 6 months and ease of locating a knowledgeable supplier or seating and mobility clinician. For the WBT group, this attitude did not change over time

The Confidence domain was based on a series of items including several asking Confidence in identifying the indications and contraindications of 17 different wheelchair components. Confidence did not change. The hands-on nature of a traditional face to face training might have been a better forum to infuse Confidence concerning the performance of different components.

CONCLUSION:

This study showed that web-based training resulted in positive changes in knowledge over time. WBT also had a positive impact on the attitude domains of Independence and Leadership. Overall, evidence suggests that web-based training has utility as a knowledge transfer mechanism. Additional psychometric development of the knowledge test and manual wheelchair questionnaire is warranted. While preliminary analysis revealed promising internal consistency and test-retest reliability, it is important to determine more fully the responsivity, validity and reliability of these newly developed measures to determine if results were due to the sensitivity of the measures or the impact of training.